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System and Method for Manipulating an Image on a Screen 
Field of Invention 

5 The present invention relates to a system and method for the manipulation of 
images on a screen, and particularly relates to a system and method for use 
with a touch-sensitive screen. 

1 0 Background of Invention 

In devices that allow a user to view an image on a screen, such as a picture or 
a diagram, a function is often provided which allows the user to zoom into or 
out of a particular portion of the image. In a typical example of such a zoom, a 
15 user must first draw a rectangular "zoom box" or "marquee" on the region of the 
screen where the zoom is required. The device then refreshes the image on 
the screen, with the region that was enclosed within the box magnified to 
occupy the whole of the screen. 

20 If the user finds that further magnification is required, then the process may be 
repeated. Thus, a box is drawn about a new region of the screen, and the 
screen is refreshed so that the new box occupies the full screen. 

There can be problems with such systems, in that in order to obtain a desired 
25 zoom, the user may need to repeat the above steps a number of times. 
Furthermore, it can be difficult to draw the box accurately enough to ensure that 
the center of the zoom is at a desired point. This may be particularly 
problematic if a number of magnifications is needed, as any error in the position 
of the actual zoom center as compared to the desired zoom center may be 
30 magnified on each step. Thus, the user may be required to repeat the whole 
magnification process should the final image not adequately show the actual 
area of interest. 
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The object of the present invention is to provide an improved system and 
method for zooming into and out of a feature of an image on a screen. 



5 Summary of Invention 

According to a first aspect of the present invention, there is provided an image 
manipulation system which includes a touch-sensitive screen and a stylus for 
indicating to a point on the screen by touching the screen, the system allowing 
1 0 a user to carry out a zoom action on an image on the screen by detecting a 
point indicated by the stylus on the screen, and by repeatedly performing a 
zoom action on the image using the detected point as the center of the zoom 
action until the stylus is removed from the screen. 

15 The zoom action may comprise an enlargement of the image on the screen 
about the zoom center and/or a reduction of the image on the screen about the 
zoom center. 

The present invention provides a system of zooming into or out of an image on 
20 a screen which is simple to use and control. It allows the user to accurately 
locate the center of the zoom, and to control the amount of zoom. Thus, the 
user can select the zoom center accurately with the stylus, can view the zoom 
action as it occurs on the screen, and can stop the zoom at any suitable point 
(the amount of zoom being defined by the amount of time that the stylus is 
25 retained on the screen). 

Typically, the screen image will be refreshed a number of times per second, 
each time being magnified or reduced by a uniform amount, and the user's 
view may be as if "flying into" the image. 



30 



Preferably, the position of the stylus on the screen is continually monitored, 
and, on movement of the stylus across the screen, the center of the zoom 
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action is altered so that the center of the zoom action follows points on the 
screen traced by the stylus. 

This feature allows the user to adjust the center of the zoom, and allows for 
more control of the zoom. This can be especially useful during the continuous 
zoom of the present invention, since, during the zoom action, as more detail or 
more regions of the image come into view on the screen, the user may become 
aware of a point of interest away from the zoom center that was originally 
chosen. 

It should be noted that it is the center of the zoom which follows the points 
traced by the stylus, and that, during the zoom, the movement of the stylus 
does not produce a "panning" effect (Panning is where the portion of an image 
that can be seen on a screen is merely moved without a change in its size). 

The present invention may be used in the display of for example pictures, 
photographs, drawings, diagrams and the like, but it is particularly preferred 
that the image to be displayed is the graphical form of a mathematical object, 
and that the image manipulation system is able to generate a graphical form of 
a mathematical object. 

The present invention may also center the image on the screen so that the 
point at which the zoom is centered is mapped towards (and preferably to) the 
center of the screen. 

This feature can be useful where the point of the image about which the zoom 
occurs is distant from the center of the screen, in which case, as the image is 
enlarged, image information adjacent to the zoom center may move off of the 
screen. The centering means may therefore move the image so that it is more 
central to the screen. This may occur automatically or on request, after a zoom 
action has finished (i.e. after the stylus has been removed from the screen), by 
e.g. mapping the image point at the center of the zoom towards the center of 
the screen. Alternatively, for example, during a zoom action, the dragging of 
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the stylus may produce a panning effect, instead of changing the zoom center, 
so that the center of the zoom can be dragged towards the center of the screen 
whilst zooming continues (The change in the function of the stylus drag could 
for example be made through the toggling of a key). 

5 

The invention also extends to a method of manipulating an image on a screen, 
and, viewed from a second aspect, the present invention provides a method of 
manipulating an image on a touch-sensitive screen using a stylus, including the 
steps of displaying the image on the screen; detecting an instruction to perform 

10 a zoom action on the image; detecting a point of contact of the stylus on the 
screen; setting a center for the zoom action at the detected point of contact of 
the stylus on the screen; performing the zoom action on the image on the 
screen using the set center of zoom; and repeating the step of performing the 
zoom action until it is detected that the stylus has been removed from contact 

15 with the screen. The zoom action may be an enlargement or a reduction of the 
image on the screen about the zoom center. 

Although the present invention is particularly applicable to a touch-screen and 
stylus device, and to their use in the manipulation of the graphical form of a 
20 mathematical object, the invention may also be applied in other manners. For 
example, instead of using a touch-screen and stylus, a digitising tablet and pen 
or puck or other suitable pointing device could be used. Also, the image might 
comprise a picture, photograph, drawing or the like, and may for example 
comprise a map. 

25 

Thus, viewed from a further aspect, the present invention provides a data- 
processing system for manipulating an image, which includes a display for 
displaying an image and an indicating device for indicating to a point on the 
display, the system generating an image on the display and providing a zoom 
30 action on the image on the screen; wherein, when a zoom action is activated, 
the system determines when the indicating device is indicating to a point on the 
screen, sets the indicated point as a zoom center, and repeatedly carries out a 
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zoom action on the image on the screen about the zoom center until it detects 
that the indicating device has stopped indicating to the point. 

Viewed from a further aspect, the present invention provides a data-processing 
5 method for the manipulation of an image on a screen, the method displaying 
the image on the screen; detecting an instruction to perform a zoom action on 
the image; detecting a point on the screen indicated to by an indicating device; 
setting a center of the zoom action at the indicated point; conducting the zoom 
action on the image on the screen about the set center of zoom; and repeating 
10 the zoom action until it is detected that the indicating device no longer indicates 
to the point. 



15 
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Brief Description of Drawings 

The invention will hereinafter be described in greater detail by reference to the 
attached drawings which show example forms of the invention. It is to be 
understood that the particularity of the drawings does not supersede the 
generality of the preceding description of the invention. 

Figure 1 is a schematic drawing of a device according to one embodiment of 
the present invention; 



Fig. 2 is a table of relations for which the zoom function is available in the 
25 device of Fig. 1; 

Fig. 3 is a view of a graphing window of the screen display showing its pixel 
and Cartesian co-ordinates; 

30 Fig. 4 is a flowchart for carrying out a zoom action on a graphical object on the 
screen; 
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Fig. 5 is a graphical view of the interactions underlying a software architecture 
that may support the zoom features of the present invention; and 

Figs. 6-12 show an example of possible screen displays of the device of Fig.1 
during a zoom procedure. 



Detailed Description 

10 Fig. 1 shows a graphics device 1 having a touch-sensitive screen 2, a key pad 
3, and a stylus 4 (shown in two positions, 4 and 4'). 

The graphics device 1 can manipulate a mathematical object, e.g. a function or 
relation having a symbolic formula, and can generate a graphical form of the 
1 5 function or relation. 

The device 1 displays the graphical form of the mathematical object to be 
manipulated in a graphing window on the screen 2, and also provides a menu 
of features on a toolbar 5 for manipulating the graphical form. A desired 
20 feature may be initiated by pointing to an associated icon 6 on the toolbar 5 
with the stylus 4. 

The touch screen 2 may take any suitable form, and may for example comprise 
a resistive, capacitive, surface acoustic wave or infrared screen, as are known 

25 in the art. The stylus may take a form appropriate for the type of touch-screen 
used. A resistive touch screen generally consists of electrically conductive and 
resistive layers separated by invisible separator dots, which, when pressed 
together cause a change in current to register a touch event. A capacitive 
touch screen has a capacitive material coating a glass surface, and oscillator 

30 circuits at its corners which vary in frequency according to where a stylus is 
placed on the screen (the stylus 4 should be wired to the device 1 to provide a 
charge at the stylus tip). A surface acoustic wave screen sends acoustic 
waves across a glass panel with a series of transducers and reflectors, and a 
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soft-tipped stylus is used to absorb the waves at the point where it touches the 
screen to register a touch event. An infrared screen includes an invisible lattice 
of infrared light beams, and detects for breaks in the beams by the stylus. 

5 In each case, the operating system of the device 1 can detect a stylus event 
and can provide the co-ordinates of the point on the screen to which the stylus 
4 points. 

Returning to the toolbar 5, two of the icons, 6a and 6b respectively, provide a 
10 dynamic zoom into the graphical form and a dynamic zoom out of the graphical 
form. 

In order to effect a zoom into the graphical form, the user first selects icon 6a 
with the stylus 4, and then places the stylus 4 on the screen 2 at the point 7 at 
1 5 which the user desires the center of the zoom to be. 

The operating system of the device notes a stylus event, and computer 
software in the device 1 which implements the present invention, continuously 
enlarges the image of the graphical form on the screen about the point 7 until 
20 the user removes the tip of the stylus 4 from the screen 2. 

If the user drags the stylus 4 over the screen 2, then the center of zoom also 
changes to follow the stylus 4. Thus, if the tip of the stylus 4 is dragged across 
the screen 2 along a path 8 to the stylus position 4\ the center of the zoom will 

25 change to follow the path, and will for example include intermediate zoom 
centers 7\7" before reaching a final zoom center 7n at the point where the 
stylus 4 comes to rest (n being the number of repeated enlargements which 
occur during the movement of the stylus between the positions 4 and 4\ which 
will depend on the rate of repetition of the enlargements and the time taken to 

30 move the stylus 4 over the path 8). 



It should be noted that the device 1 continues to zoom into the image on the 
screen 2 during the changes in the location of the zoom center 7. 
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The device 1 thus allows a user to zoom into a graphical form on the screen 2 
in a continuous manner using the stylus 4, so that the user can simply and 
easily control the amount of zoom through the amount of time that the stylus 4 
5 is held in contact with the screen 2. Further, the user is easily able to control 
the center of the zoom during zooming so as to remain in the area of interest. 
This latter feature is particularly advantageous in that the area of interest to the 
user may change during the zoom as for example more of the detail of the 
graphical form is revealed. 

10 

As well as zooming into the graphical form, the user may zoom out of the 
graphical form by selecting icon 6b, and by then placing the stylus 4 on the 
screen 2 at the desired center of the zoom-out until the desired amount of 
zoom-out has been achieved at which point the user removes the stylus 4 from 
15 the screen 2. 

Generally, a number of other manipulations are also possible besides zooming, 
and these may include: "panning" of the co-ordinate plane (i.e. changing the 
rectangular portion of the infinite co-ordinate plane that is visible on the 
20 screen), and translating or dilating a function. 

The zooming into and zooming out of the graphical form may be allowed for a 
specific collection of relations that are built into the device 1 . Such relations 
may for example take the form of the table of relations shown in Fig. 2. Instead 

25 of choosing a relation from a pre-defined list, the device 1 may also allow a 
user to define the mathematical relation by inputting desired information into 
the device in order to construct a mathematical equation. In this case, the 
device 1 may include a software component for classifying the user-defined 
relation as conforming to one of a set of pre-defined relations built into the 

30 device. 

The process of zooming into or out of a graphical form involves the 
recalculation of the portion of the Cartesian co-ordinate plane of the graphical 
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form that will be visible on the screen (which will depend on the zoom factor 
and on the center of the zoom), and the recalculation of the values that the 
graphical form will take in this portion of the Cartesian plane. 

5 Referring to Figure 3, the graphing window of the screen 2 in which the 
graphical form will appear consists of a rectangular grid of pixels that measures 
p x pixels horizontally and p y pixels vertically, and has an upper left corner with 
pixel co-ordinates of (0,0), and a bottom right corner with pixel co-ordinates of 
(Px-1, Py-1). It also has Cartesian co-ordinate boundaries of (x min» ymax) and 

1 0 (X m ax» ymin)- 

li When a zoom into the graphical form occurs, the scale of the Cartesian co- 

| y ordinates is increased in accordance with the zoom factor, so that a smaller 

Z" region of the graphical form is seen on the screen in greater detail. Thus, if the 

11 15 distance between each row of pixels on the screen 2 before a zoom is Ay 
I Cartesian units apart, and the distance between each column of pixels before a 

^3 zoom is Ax Cartesian units apart, then, after a zoom into a graphical form by a 
factor of Z, the distance between the screen pixels, as measured by the 

V* rescaled Cartesian co-ordinate system, will be Ax/Z and Ay/Z (since the scale 

H 20 of the Cartesian co-ordinates will increase by a factor of Z). 

As can be see from Fig. 3, a zoom into or out of the window by a factor of Z 
about a pixel point with pixel co-ordinates (h,v) and Cartesian co-ordinates 
(x c ,yc) would provide new screen boundaries having Cartesian co-ordinates of: 

25 

Xmin = x c - Ax*h, x max = Xc + Ax*(p x - 1 - h) 

ymin = y c - Ay*(p y -1 - v) y max = y c + Ay*v (Eqn. 1 ) 



where Ax and Ay are the new distances between the pixels 
30 values, e.g. increased or decreased from their previous values by the 

zoom factor Z. 



Attorney Docket No. 70006210-1 



10 



10 

The zoom factor may be suitably small (close to 1 ) so that the user's perception 
of the zoom is that it is continuous. 

Figure 4 shows a flowchart for conducting a zoom into or out of a graphical 
form that is displayed on the screen 2 in accordance with the present invention 
which may be implemented by a software application running on the device 1 . 
It will be understood that, in the light of the present teachings of the invention, a 
programmer may implement the flowchart of Fig. 3 as software in a variety of 
manners. 



In step S10, the zoom process is called when a Zoom Icon event is noted (the 
selection of either icon 6a or 6b). The process then passes to step S20, in 
which a check is made for a stylus event on the screen. When such an event 
occurs, the process passes to step S30, where the pixel co-ordinates (h,v) of 

15 the point indicated by the stylus are determined from the stylus position on the 
touch screen given by the device's operating system. Next, in step S40, the 
process determines the point (x c , y c ) in the Cartesian co-ordinate plane of the 
graphical form which corresponds to the pixel point (h, v), and, in step S50, 
obtains the zoom factor. The zoom factor will be stored in memory, and is 

20 preset, but, if desired, may be varied by the user prior to zooming. 

The process then passes to step S60, in which the new Cartesian co-ordinate 
screen boundaries are calculated in accordance with Eqn. 1 above, and to step 
S70, where the graphical form is replotted for the domain of the new screen 
25 boundaries. The new graphical form plots are then displayed on the screen in 
step S80. 

Steps S20-S80 are repeatedly performed until the user removes the stylus 4 
from the screen 2. Also, as the stylus 4 moves across the screen 2, so does 
30 the center of the zoom. The routine of Fig. 3 may end when the user exits the 
zoom feature by for example selecting a different feature from the toolbar 5. 
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The replotting and drawing of the graphical form in steps S70 and S80 may 
involve the determination of a value for the underlying relation for each possible 
x-axis value of the Cartesian co-ordinates that may be displayed on the 
graphing window, or may involve the determination of values at only a few 
5 selected points, and then the drawing of the graphical form on the screen using 
an interpolation method or other line drawing technique. Such interpolation 
methods would be selected depending on the function to be drawn. Thus, each 
relation listed in the table of Fig. 2 may have an interpolation method 
associated with it. This method may be determined in accordance with the 
10 known mathematical properties of the relation, as would be understood by a 
person skilled in the art. 

If more than one graphical form is displayed on the screen, then the 
calculations are carried out for all of the graphical forms, and all of the graphical 
1 5 forms are re-displayed. 

As well as the zoom action, the device 1 may allow the re-centering of the 
image on the graphing window of the screen 2, so that for example after the 
zoom action has finishes, the center of the zoom action may be mapped to the 
20 center of the graphing window. 

An underlying software architecture that may support the zoom feature of the 
device 1 and which uses an object-oriented methodology is shown in Fig. 5. 

25 In Fig. 5, the Relation is a software component which stores the formulae 
representing the mathematical relations which are shown on the screen 2 of the 
device 1. 

The Plotview is a software component which is responsible for the display of 
30 the graphical form of the underlying Relation objects and for the processing of 
the user inputs which use the stylus in the graphing window of the screen 2 
(e.g. translation, dilation, tracing and the like). It interacts with the PlotControl 
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component to which it passes user interactions and from which it receives 
notifications about changes in the Relations. 

The PlotControl is a software component which facilitates the exchange of 
5 information between the PlotView and the Relation object that the PlotView 
represents. In the direction of PlotView to Relation, the information represents 
changes to the Relation (translation, etc.), or requests for recalculations (such 
as those triggered by a change in the screen boundaries caused by a zoom 
action). This step involves the PlotEvaluationAlgorithm. In the direction of 
10 Relation to Plotview, the information represents the data necessary to display 
the graph. 

%j 

^ The PlotEvaluationAlgorithm is a collection of mathematical rules and 

a S procedures used to efficiently produce outputs for a particular relation over the 

\% 15 domain of the values on the graphing window. Thus, the procedures may 
In provide for the calculation of only a few of the graphical form points in the 

IU domain to be displayed, and may provide an interpolation algorithm for drawing 

-« the remainder of the graphical form based on the known mathematical 

f[i properties of the relation. Each relation of the table of Fig. 2 will for example 

P 20 have its own dedicated algorithm. 

When a user initiates a command to zoom in to or out of a graphical form on 
the screen 2, a "Graph View" component, which controls the screen display first 
determines the type of zoom (based upon the icon pressed, either 6a or 6b) 
25 and the location of the stylus tip. The PlotView then uses the horizontal and 
vertical zoom factors to recalculate the horizontal and vertical co-ordinates of 
the screen boundaries. The Graph Settings (i.e. the maximum and minimum 
co-ordinates for the horizontal and vertical boundaries of the screen) are then 
updated, and this triggers the calculation of the plots. 

30 

For each Relation that appears in the PlotView, the following occurs: The 
PlotView notifies the PlotControl object for the current relation of the change in 
the screen boundary values. The PlotControl then uses the 
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PlotEvaluationAlgorithm to recalculate the plot. The PlotEvaluationAlgorithm 
interrogates the Relation for information needed (such as the parameters of the 
relation) to recalculate the plot. Once the plot is calculated, the PlotControl 
sends the new plot to the PlotView for display. 



This entire process of determining the zoom center, recalculating the Graph 
Settings and redrawing all of the graphical forms is repeated continuously while 
the stylus 4 is held on the screen 2. 

10 Although the above embodiments have dealt with the display of a graphical 
form of a mathematical object, the invention may also be applied to other 
images. For example, the displayed image may be a drawing, picture, 
photograph, diagram or the like, and, could, for example, comprise a map. In 
such applications, the action of zooming into or out of the image may take the 

15 form of determining a new portion of the image to view based on the zoom 
factor and zoom center and resampling a bitmap of the image stored in the 
device's memory to provide the new portion of the image on the screen in 
greater or lesser detail. 

20 Also, instead of using a stylus device, the invention may be applied with any 
other suitable apparatus. For example, the invention may use a digitising tablet 
and pen or puck, a light-pen, or some other suitable pointing device, to 
determine the center of the zoom. 

25 It is to be understood that various alterations additions and/or modifications 
may be made to the parts previously described without departing from the 
ambit of the invention. 

30 Example 



5 
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An example of screen displays which might be provided during a zoom into a 
graph of a standard sine function, y = b*sin (a*(x-h)) + k, of the table of Fig. 2 is 
shown in Figs. 6 to 12. 



5 Thus, the first display of screen 2 (Fig. 6) is clear to start with. The user then 
chooses a sine function from the options provided by a dropdown menu in Fig. 
7, and edits the symbolic form of the function as desired in Fig. 8 (the user may 
change any of the parameters a, b, h or k in the standard function). 
Alternatively, the device 1 may provide the user with the option to input 
10 information to define their own function, in which case, the device may include 
a software component to identify the user-defined function as being one of a 
pre-defined set of relations as in Fig. 2. 

Once this is done, the device 1 presents the symbolic and graphical forms of 
1 5 the mathematical function on the screen 2 as in Fig. 9. 

The user can then zoom into the graphical form of the function by choosing the 
zoom-in tool 6a from the toolbar 5, and by placing and holding the stylus 4 on 
the screen 2 at the desired center of zoom. In this example the user should be 

20 considered to have placed the stylus 4 at the origin of the graphical form, and 
so the zoom is centered on the origin also. This produces a continuous zoom 
into the graphical form, with a continuous progression of incremental 
enlargements (including the instantaneous screen shots of Figs. 10 and 11) 
being shown on the screen 2 until the user lifts the stylus 4 from the screen 2 to 

25 leave for example the screen display of Fig. 12. 

In the case of a zoom out of the graphical form, the user would select the 
zoom-out icon 6b, place the stylus 4 on the screen 2 at the desired center of 
the zoom-out, e.g. again at the origin, and the screen display would then 
30 continually reduce the image, for example beginning at Fig. 12 and ending at 
Fig. 9 when the stylus 4 is removed from contact with the screen 2. 
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